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The fundamentals of the theory of denurnerable Markov chains with stationary transi- 
tion probabilities were laid down by Kolmogorov, and further work was done by Doblin. 
The theory of recurrent events of Feller is closely related, if not coextensive. Some new 
results obtained by T. E. Harris turn out to tie up very nicely with some amplifications of 
Doblin's work. Harris was led to consider the probabilities of hitting one state before 
another, starting from a third one. This idea of considering three states, one initial, one 
"taboo", and one final, is more fully developed in the present work. The notion of first 
passage time to the "union" or "intersection" of two states is also introduced here. The 
interplay between these notions is illustrated. 



The fundamentals of the theory of denurnerable 
Markov chains 2 with stationary transition proba- 
bilities (DMCS) were laid down by Kolmogorov 
[1] 3 and further work was done by Doblin [2], The 
theory of recurrent events of Feller [3] is closely 
related, if not coextensive. Recently some interest- 
ing new results were obtained by T. E. Harris [4] and 
communicated to the author. They turn out to tie 
up very nicely with some amplifications of Doblin's 
work the author was engaged in. Although Harris' 
main purpose lies elsewhere, he was led to consider 
the probabilities of hitting one state before another, 
starting from a third one. This idea of considering 
three (instead of the customary two) states, one 
initial, one " taboo," and one final, will be more fully 
developed in the present work. The notion of first 
passage time to the "union" or " intersection" of 
two states will also be introduced here. The inter- 
play between these notions will be illustrated. 

Recorded results in this paper will be labeled as 
formulas and theorems, respectively. Relevant re- 
marks as to their origin or significance will be found 
in the body of the paper. The author is indebted to 
Dr. Harris for communicating some of his results 
before publication. 

1. The sequence of random variables {X n }, n = 0, 
1,2, .. . forms a DMCS. The states will be de- 
noted simply by the positive integers. The (one- 
step) transition probability from the state i to the 
state j will be denoted by Pff . Because of station- 
arity we have 

ptr=p(x m+1 =j\x m =i) 

for all integers m>0 for which the conditional prob- 
ability is defined. With this understanding, we shall 
permit ourselves to write m=0 in the definitions to 
follow, as if the conditional probabilities were always 
defined. 

Notations: 

ft, N, v, r, s, denote positive integers and will be 
used as time parameters or general numerals; 
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2 "Denurnerable" means "with a denurnerable number of states;" "chain" 
refors to a process with an integral time parameter. 

3 Figures in brackets indicate the literature references at the end of this paper. 



i, j, k, I, h y denote positive integers and will be 
used as state labels: 



Pir=P(x n =j\x =iy, Pff: 



;o, »Vj 

k P>f=P(X n = j, X v *k, l<v<n\X =i) 
F<f = P(X n =j, X,*j, l<v<n\X =i) 
k F!f=P(X n =j, X v *j,*Jc, l<v<n\X =i) 



Q* 



it l 



where Q may stand for any of the symbols k P tjJ F ij7 



or k Fij. 



to the above notations. 
'passage"; the letter F, 



We offer the following clue 
The letter P designates 

"first passage"; the first right-hand subscript desig- 
nates the initial state; the second, the final state; the 
left-hand subscript designates the "taboo state," 
namely, one to be eschewed during the passage (ex- 
clusive of both terminals); the star on a letter with 
subscripts designates the sum of the corresponding 
infinite series (finite or + ») summed from ft=l ad 
inf. We admit that this is not the most logical 
system of notations we could have invented. For 
instance, we have the superfluity F^ ) = j P^ ) J and if 
we had allowed more than one left-hand subscript, 4 
we could have used only one letter P and written 
k Flf= jyk Pif. However, we consider our notations 
to be preferred to the arbitrary use of all sorts of 
letters from the Latin and Greek alphabets. Also, 
after painful deliberations we decided not to define 
fcP/f , Fff , or k Fl f , while reserving the right to do so 
later in some cases. 

Formula I: If i^j, then 

where on the right side 0- oo is to be taken as 0. 5 

* This naturally suggests the consideration of more than one taboo state. 
s This follows also from the easily interpreted relations 

n=0 l—i*U i*H 

The convention that 0- oo is to be taken as will be understood in similar cir- 
cumstances. 
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Proof: We start from the formula 

n-l 

Win) -ST-\ p(v) JPin-v) 

± a Z-j i x a i L a j 

r-0 



(i) 



where we agree that ^P/J ) = l. Equation (1) is 
proved as follows. Either the state i is not entered 
at all during the passage from i to j, which contin- 
gency contributes the term corresponding to #=0 on 
the right side of (1); or there is a last entry of i, 
occurring at the #th step, l<v<n—l, which con- 
tingency contributes the general term. 
Summing (1) over n y we obtain 



n = \ n=l v=0 



0=0 n=v-\-l 



=«w +,pa). 



Since the terms of the double series are nonnegative, 
the inversion is justified and (I) is proved. Moreover, 
this proves that if *^*>0, then *P*< °° . It follows 
from (I) that t F£=Q if, and only if, P*=0, namelv, 
Pft^Oforafln. 



Formula II: If j^k, then 

*P*.=*F*.(1+,P*-). 



(ID 



(This formula is easily interpreted in terms of math- 
ematical expectations.) 

Proof: We start from the formula 

•P£'=s **■# fn-\ (2) 

v = l 

where as before k P}f = 1 . If we ignore the left-hand 
subscripts, (2) reduces to a familiar formula. The 
proof of the latter extends immediately to (2). 
Summing (2) over n we obtain 



n = \ n = l v=\ 



v=l n=v 

We note the following corollaries to (I) and (II), 
to be used later. 

Formula Ha: If i^j, then 

t P*. = ,P*.(l + ,P*). (Ha) 

Formula lib: If j^k, then 

j? k j>? t =nJ7i. (nb) 

Formula He: If i^j, then 

P*(1+,P*)=^(1+^). (He) 



Formula III: If i ^j, then 



N 



s pir 

lim B=0 



n=0 



(HI) 



Proof: We start from the formula 

n 

p. (n) = , y^ pi*) .p^~ v) 



(3) 



where we agree that < P/f=0. The proof of (3) is 
entirely similar to that of (1). 

Summing (3) from n = to n=N, we obtain 

N N n N N-v 

Sf/^E s Pu ^/r e) =s p& s W. (4) 



We need an elementary lemma which is frequently 
useful in such connections. 

Lemma. Let 0<a v <l, b v >0; X) «„>(), lim b v = 



5<+oo. Then 



-B. 



t?=0 



Applying the lemma to (4) we obtain (III). That 
t P*< oo is clear from (Ha), and the remarks at the 
end of the proof of (I). 
Theorem 1. The limit 



lim 

iV->oo 



N 

S p if 

n=0 
N 

S P^ 

n=0 



(5) 



exists, and is equal to any of the following three 
expressions: 



i+jP*i n_ F? iS F? i m 

1+iPf/ <P£' ^SfF*' 



(IVa,b,c) 



the first always, the second if i 7±j, the third if F*F*i >0. 
Proof. Doblin [2] has shown, trivially, that 



N 

lim 2^- =Ff,. 

71 = 



(6) 



Comparing (III) and (6), we obtain (IVb) if i^j. 
(IVa) now follows from (lie) and obviously holds for 
i=j. If F?F*i>0, then the denominator of (IVc) is 
not zero, and this is then equal to (IVb) by (lib), 
with k=i. 
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That the limit (5) exists, and is finite and not zero, 
was proved by Doblin [2]; that it is equal to (IVa) 
was previously proved by the author [5]. The 
present approach seems to be the simplest. 

Corollary. If i,j,k,l, are distinct states of one 
class,* then 

N 

£r " _jy,(i+,p;,) 



lim 

iV-xx 



N 



S^' 



-F*(l+jP*) 



71 = 1 



Naturally there are other expressions for it, and 
we omit the tedious considerations when some of the 
states are identical. 

2. We now consider two states i and j belonging 
to the same recurrent class, namely: 

v* _ jf* = f* = i 

•*■ it M i j x • j x • 

A fundamental idea in the theory of DMCS, already 
found in Kolmogorov's work, is that whatever tran- 
spires between successive entries at a recurrent state 
forms a sequence of independent events. Using this 
idea, Harris [4] and Levy [10], independently of each 
other, discovered theorem 2. Our proof is somewhat 
different from theirs. 
Let i 7^i and define 



r" n =the number of v, l<v<n, such that X v 
Z„=the number of v, l<v<n, such that X v - 






In words, Y n (or Z n ) is the number of entries at the 
state i (or j) in the first n steps. Using the average 
ergodic theorem (see (11) below) it is easy to show 
that if j is a positive state, and P(X eC) = l, where 
C is the class containing j, then we have 



^lim-^-=l\=l. 

^ f=0 J / 



The following theorem covers both positive and null 
classes. 

Theorem 2 (Harris-Levy) . // i and j are two states 
in a recurrent class C and P(X eC) =1, then 



1*^ n i • v = 
im r -=hm-^ r 



sn" 



2W 



(7) 



Proof. 

+ «)=!. 



Since i is recurrent, we have P/lim Y n = 

\n— >°o 

Let Vi<^v 2 <^ ... be the successive in- 
dices Vj such that X v =i. Let W r s =the number of 
v, v s <Cv<^v s+ i such that X v =j. Then as remarked 
above, the W s 's are independent, identically dis- 

6 Slightly generalizing Kolmogorov, we define a class to be a set of states such 
that for any two states i and ; belonging to it, we have F*-F* { >0. See [6]. 



tributed random variables. Evidently we have 

71 = 1 

Now by definition we have v Y <n<^v Y +1 and 



sW,<z*<< 

5 = 1 


5=1 


Consequently, 




F„-l 

s=l ^ ^n ^ 

Y ~ Y — 


vi , 7=1 
Y Y 



(8) 



Applying Khintchine-Kolmogorov's strong law of 
large numbers (see, e. g., [9] p. 208) to the sequence 
{ W 8 } we obtain: 



(9) 




= .P* 1=1 



Moreover, P(^< + oo) = l. It follows from (8) and 
(9) that 

pAimf^PsW (10) 

Now F* = l. Hence theorem 2 follows from (10) 
and theorem 1, using (IVb) there. 

This theorem includes as special case a previous 
result by Erdos and the author [7]. Consider in- 
dependent, identically distributed random variables 
{ U n } which assume only integer values with mean 
zero. They form a DMCS with all integers as the 
states. Since the mean is zero, all possible states 
are recurrent by a theorem of Fuchs and the author 
[8]. 7 Without loss of generality, we may suppose that 
every integer is a possible, therefore recurrent, state. 

n 

Writing S W = ]C U v , we see that 

v=l 

py=P(S n =j-i). 
Hence, P$ > =P<fi ) =P$ ) \ and (7) becomes 



p(limfs=lW, 



which is theorem 8 in [7]. Needless to say, as far as 
this statement is concerned, Harris 7 approach is 
incomparably better. However, we note that there 
we actually proved a sharper result, i. e. 



(^ 



-^>M„ 4 i.o.)=0 

n / 



7 This important step cannot be circumvented by the present, more general, 
approach. 
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for every e>0, where M n =^ P(S v =i). See also 

v = l 

theorem 7 in [7]. It would be of interest to investi- 
gate corresponding strong relations for the general 
Markov-chain case, using perhaps a more precise 
form of the strong law of large numbers. 

3. We now consider a positive recurrent class C. 
According to Kolmogorov, in C all mean recurrence 
and first passage times are finite, namely, for all 
i,jeC we have 

n=l 

We introduce the notions of first passage to j\Jk and 
to jf\k, as follows. Let j^k. 

T(i,j\Jk) = ihe smallest integer n>\ such that 
X n =j or X n =k, whichever happens first, given that 
X =i; m 

T(i,jf]k) = the smallest integer n>\ such that 
there exist two integers n x and n 2 such that ni^n 2 , 
1<Wi<ti, 1^^2^^, and X ni =j, X n2 =k, given that 

°m(i,j\Jk)=E{T(i,jl)k)}; 
m(i,jnk)=E{T(i,jrik)}. 
Let w denote the sample point. Put 

e' n = { w:X (w)=i, X v {w) ^j,^k 

for 1 <v<n;X n (w)=j}; 

e j = U el. 

71 = 1 

Thus e j is the event that X =i and the state j is 
reached before the state k. Since i, j, and k belong 
to one recurrent class, we have 

e j \Je k = {w:X (w) = i} . 

Let 'P(Xq=i)=c^>0. We have the following rela- 
tions, immediate consequences of the definitions. 

cm(i,j\Jk) = f^i I nP(dw) + f nP(dw) 

n = 1 ' J e } n J e * 

«=S{ f nP(dw)+ ( (n+m ki )dP(w)\ 

n = l \Je\ Je K n ) 



cm 



= cm(i, j\Jk)+P(e k )m kj 
cm(i,r\jk)=Jl\ (n + m jk )P(dw) 

n = l {Je[ 

+ jjn+m k) )P(dv>)} 

=cm(i, jU*)+P(«On» # +P(e*)in w . 
Now by definition we have 



P(e>) 



k 1 tj) 



P(e k ) 



J 1 ik' 



Hence we obtain from the above: 
Formula VI. If j^k, then 

m(i, j\Jk)=mij— j Ff k m kj =m ik — k Ff i m jk . (VI) 
Formula VII. If j^k, then 

m(i, jnk) = m ij + k F* i m jk =m ik + j F* k m k1 . (VII) 
Since 

»FS +,*?,= 1 

we deduce from (VII): 

Formula VIII. If j^ k, then 

7n ik +m kj — m ij = k Ff i (m Jk +m k j). (VIII) 

We note the following special case (i=k) of (VIII): 

m kk = k F^(m jk +m kJ ). (Villa) 

This last formula is due to Harris [4], who also de- 
rived from it the following relation: 



m ji=j Ff k 

nikk kFti 



(VHIb) 



Now in a positive class the ergodic theorem of Kol- 
mogorov holds: 8 



1 n 1 

n— »oo n v=i Wlj; 



(ii) 



Thus (VHIb) turns out to be a special case of 
theorem 1, using (IVc) and noting that Ff k =F ki = l. 
Dividing (VIII) by the product of (Villa) and 
(VHIb) we obtain 



mik+rrikj-rntj 



m j 



J 1 ik 



(12) 



By formula (lib), the right side is k P* ; , since Ff k = 1 in 
the present case. Thus we obtain 
Formula (IX). If jy^k then 



P* 
. a- 



mu+mjcj-mij 



Mj 



(IX) 



As an application consider, as in the Central Limit 
Theorem for Markov chains, random variables { Y n } 
attached to the Markov chain { X n } in the following 
way: Y n =Xi if X n =i where the xjs are arbitrary 
real numbers. 

Theorem 3. Let ibe a positive state. Given X =i, 
let v denote the smallest n>l such that X n =i. Then 
if the series on the right-hand side converges absolutely, 



8 This theorem actually establishes the limit of P[f as n— »co . The average form 
(11) is an easy consequence of a Hardy-Littlewood Tauberian theorem. 
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we have 



Xj 



E{Y 1 +. . .+Y VQ \Xo=i} = m ii T; r 

j = l IILjj 

*#{(F 1 +...+F„ ) 2 jZ =i} 



(Xa) 



j = i m^ j = i mjj fc = 1 m kk 

(Xb) 

Phoof. It is more convenient to consider new 
variables Z n denned as follows: 



Z n 



if X v =i for some v, 1 <v<^n; 
Xj if X n =j and X^i, 1 <#<>, 



where j may be i in the last-written line. Evidently 
we have then 



(n = l 



=T,E\z n X =i\ = ±T li P!l ) x J 

n=\ \ ) 7i = \ j=l 



by (IX) with i=k. 
Furthermore, we have 



i=i j=i ^j 



*{C£ r ') , M-*{(£*) , M 

(n=l\ l<r<s<n / 

As before, we obtain readily 



Next we have 



00 00 



r = i s = r-hl 

00 00 00 CO 00 00 

r =U = r+l ; = 1 fc = l j = \ k = l 

Mi Mi 

By (IX) this reduces to (Xb). 

The two expressions on the left sides of (Xa) and 
(Xb) play an important role in Doblm/s Central 
Limit Theorem for DMCS. We refer the reader to 
[2] for details. They are here evaluated in what 
seems to us more tangible terms. 

4. From formulas (VI) and (VII) it follows that 



9 Doob pointed out to me that this is a case of W aid's equation for Markov 
chains. 



m(ij[Jk) +m(i,j(Vc) =m ij +m ik . 

This relation is in striking resemblance to a familiar 
formula in the elementary calculus of probabilities, 
according to which if A and B are any two events 
then 

P(A\JB)+P(AC\B)=P(A)+P(B). 

The generalizations of the last relation to any 
finite number of events is known as Poincare's for- 
mula (see, for example, [9], p. 61); and we immedi- 
ately suspect that the same may be true for the mean 
first passage times. This is indeed so. We define 
m(i,ji\J. . .\Jj s ) and m^Jifl. . .f\j s ) as the obvious 
extensions from the case s = 2. We shall also write 

.7i n . . .nj 8 ^-j to denote jiD. . .rv r _irVr+in. . ny s if 

j=j r (l<r<s) and jifl. . .Dj s ii j is not one of the 

Fohmula XL If ji r . .,j s are distinct states in a 
positive class to which i also belongs, then 

s 

m(i, jiU . . . Uj s )=J2 m(i, j r )— J2 ™>(i, hS^h 2 ) 



+ 2 

l<ri<r 2 <r 3 <s 



Proof. Put 



r=l l<ri<r2<s 



+ 



(-iy- 1 m(i,j 1 r\.,.Cij 9 ). (xi) 



e r n = {w:X (w) = i, X v (w)j*j u . . . , j s 

for 1 < v<n, X n (w)=j r } 
e T = U e r n . 

n = l 

We have, as at the beginning of section 3, 

™(hj ri n • . • nir,) 

^c" 1 SS f {n + m(j r ,j ri n . . . r\jr i +-jr)}P{dw) 
= m(i,j 1 \J . . . Uj 8 ) 

+ J2c- l P(e')m(j r , j rt n . . . Hi, -*-i). (13) 

r = l 

Substitute (13) into the right side of (XI) and 
consider the typical term c~ l P(e r )m(j r ,j ri C\ . . . 
fY/rj), where r is distinct from r h . . . ,r z . It 
appears on the right side of (XI) once in m(i,j r C\ . . . 
r\j Tl ) and once in m(i,j r nj ri r\ . . . flj^), with 
opposite signs; and does not appear in any other 
terms. Hence its net coefficient on the right side 
of (XI) is zero. It remains only to consider the 
term m(i,jx\J . . . \Jj s ). This appears once in 
every term and hence its net coefficient is 

(0-0)+(0- ■ "K-i^-i- 

Therefore (XI) is established. 
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We remark that trivial as this proof is, it does not 
exactly correspond with the familiar proof of Poin- 
care's formula, and we do not know if there is any 
closer relation between the two apparent twins. 
We also leave possible extensions suggested by the 
known extensions of Poincare's formula to the 
interested reader. 

5. We now give another method of computing 
k P*. This method requires the ergodic theorem 
(11). An interesting byproduct is the following: 

Theorem 5. Ij i, j and k (not necessarily distinct) 
belong to a positive class, then 



S {Ptt-P? 



m 



jk - 



-Vdi 



m kk 



Proof. Using the familiar formula (cf . the remark 
after (2)) 



we have 



N 



SW 



P(n) X~^ F(»)P(»- 
ik — ' ' L ik 1 kk 
= 1 



-p% ) -s {Fn-Fn } if piv. a*) 



Substituting from (11), we see that the right side 
of (14) is, as N-> <*> , asymptotically equal to 



N 



F%- 



-Ffk 



N-y f 
m kk ' 



(15) 



-1, HvF?l = m ik <co, ^vFfr 



Now since 

•*■ ik x J 

we have, as N— > oo 

* jk 



V = l 



--N S {F%-Fft\ 



= m ifc <c 



>0. 



Using this in (15), we see that its limit as iV— ><» is 

_m jk —m ik 



1 N 
lim — S (- 

N-><*> m<kk v=i 



•v){F$' 



-Fn 



m kk 



We note that theorem 5 gives a convenient de- 
termination of the mean first passage times in terms 
of the transition probabilities; in particular 

"»i»=(l + S {PIV-PIV })lim ^ S PiV- 

We do not know what the situation is in a null class. 
All we can infer from theorem 1, (IVc), is that if 
i and j belong to one recurrent class and jFf^iFf^ 
then 



To return to k P*. 



If jV&, we have evidently 

(16) 



X^V pin) p(l) p(»+D 

/ i * x ih L hi — k 1 ij y 



where, as later, an unspecified summation runs from 
1 to oo. Summing (16) over n, we obtain 



x-v p* pa). 



or 



since 



^ta p* pa) p* i pa) 

' ' k r ih L hi — k r a \ r ki 
h 



p* pa) 

1 a k-L a y 

pa) 

j. ij 



(17) 



T p*_i 7 pa)_pa) 
k 1 ik — L i k r a — i a • 

We assert that in general 



This is readily shown by induction on n, starting 
with (17). Now sum from n— 1 to n=N, divide by 
N, and let N — > » . By (1 1 ) and theorem 5 we obtain 

(s *n)^r == * p «+s W-Pff } 



*P«*,4 



niij—m 



*j 



m* 



(18) 



Now, 

A /l=l W = l 

=SP(^.^A,l<»<n|^o=i)=S S^'=% 

71=1 71 = 1 v = n 

(19) 
(18) and (19) give (IX). 
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